Spin-glass phase transitions on real-world graphs 
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We use the Bethe approximation to calculate the critical temperature for the transition from a 
paramagnetic to a glassy phase in spin-glass models on real-world graphs. Our criterion is based 
on the marginal stability of the minimum of the Bethe free energy. For uniform degree random 
graphs (equivalent to the Viana-Bray model) our numerical results, obtained by averaging single 
problem instances, are in agreement with the known critical temperature obtained by use of the 
replica method. Contrary to the replica method, our method immediately generalizes to arbitrary 
(random) graphs. We present new results for Barabasi- Albert scale-free random graphs, for which 
no analytical results are known. We investigate the scaling behavior of the critical temperature 
with graph size for both the finite and the infinite connectivity limit. We compare these with the 
naive Mean Field results. We observe that the Belief Propagation algorithm converges only in the 
paramagnetic regime. 

PACS numbers: 75.10.Nr, 89.75.-k 
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Sparse networks with non-uniform topology occur 
in many contexts ranging from biology and sociology 
to communication systems and computer science 
Prominent examples of mathematical models designed to 
capture the topological features of these so-called "real- 
world" graphs are uniform degree random graphs 0, 
scale- free networks and small- world graphs p| . While 
much research has been done on topological properties 
of these complex networks, not much seems to be known 
about the properties of such networks when nodes con- 
tain dynamic variables and links represent positive or 
negative interaction between them. Examples of complex 
networks with interacting nodes are networks of neurons 
in the brain communication in networks of sensors 
0, combinatoric optimization problems Q and protein 
folding 0,0. In this Letter, we take a statistical physics 
approach to study the behavior of such systems. 

We study generalizations of the SK model where 
we replace the fully connected underlying graph by com- 
plex random graphs. We use the Bethe approximation, 
an extension of the naive Mean Field (MF) approxima- 
tion, to calculate the critical temperature correspond- 
ing to the phase transition from a paramagnetic to a 
spin-glass-like phase. The transition is characterized by 
a marginal instability of the Hessian of the Bethe free 
energy. For uniform degree random graphs, averaging 
over the interactions and the underlying graph instances 
yields results consistent with known analytical results ob- 
tained by use of the replica method • Contrary to the 
replica method, our method immediately generalizes to 
arbitrary (random) graphs. We present results for scale- 
free networks, for which no analytical results are known 
to the best of our knowledge (however, for the ferromag- 
netic case on random graphs with arbitrary degree dis- 
tribution, see |12j). 

Another advantage of the Bethe approximation is that 
it yields results for finite TV and for specific problem in- 



stances of the interactions. This is important for techno- 
logically relevant applications viz. image restoration |l.'i| , 
artificial vision decoding of error-correcting codes 
[lfij and medical diagnosis 0. The problems in these 
applications can generally be reformulated in terms of 
thermodynamic systems defined on graphs, and solving 
them amounts to the determination of the Boltzmann 
distribution for these systems. 

Let G = (V, B) be an undirected labelled graph with- 
out self-connections, defined by a set of vertices V = 
{1, . . . , N} (corresponding to the spins) and a set of edges 
B C 1 1 < i < j < N} (corresponding to non-zero 

interactions between the spins). We define the adjacency 
matrix M corresponding to G as follows: My = 1 if 
(ij) E B or (ji) G B and otherwise. Denote by Ni 
the set of neighbors of vertex i, and denote the degree 
(connectivity) of vertex i by d{ :— |A^| = ^ g7 Afij. 

We define the average degree d := ■h ^2 ie y di and the 
maximum degree A := max^gv di. 

To each vertex i € V we associate an Ising spin Sj, 
taking values in { — 1, +1}. Let J be a symmetric N x N 
matrix representing the interactions, which we do not 
further specify for the moment, except for the constraint 
that J should be compatible with the adjacency matrix 



M, 



Ji', 



if Mi. 



0. For the Hamiltonian of the 



system we take 



H = 



E 



JijSiSj 



and we study the corresponding Boltzmann distribution 
over the configurations s = (si, . . . , sn) € { — 1, +1}^: 



1 




■ exp (3 



Note that, because of the sign reversal symmetry, the 
exact magnetizations are given by (sj) = 0. 
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The Bethe free energy can be written 0| as a 
function of the parameters m = (mi, . . . , m^r) and £ = 



^BeKO :=-/3 



A' 



i=i 

E E » 

(ij)efl Si,s 3 =±l 



E »? 

Si=±l 



1 + ra^ 



where r){x) := xlogx. The Bethe approximation consists 
of minimizing the Bethe free energy with respect to the 
parameters m and £ under the following constraints: 



- 1 < m, < 1 

- 1 < &j < 1 

1 + miO + m-j(j' 



&,W > for all a, a 1 = ±1. 



The parameters m t are then approximations for the 
means (s,-), while the £y are approximations for the mo- 
ments (si-Sj) for (z7') € £?; if G is a tree, these approx- 
imations are exact 19] . It is straightforward to check 
that TOj = and £y = tanh(/3Jy) is a stationary point of 
-Fse, which we will call the paramagnetic solution. For 
this solution to be a minimum, the Hessian of Fs e has 
to be positive definite. Defining := £y for € £> 
and £y := if £ B and (j, i) £ B, this is equivalent 
to the matrix 



(-4 



Be lik 



d 2 F Be I 

dmkdrrti lm=0,f=tanh(/3.7) 




S,ik 



(1) 



1-& 



being positive definite, since the dFsel 1 dm^d^ij part van- 
ishes and the other block dF^el 'd£ijd£ki is positive defi- 
nite for all possible parameter values. 

We will compare the results with the naive Mean Field 
approximation. The MF free energy is given by 



Fmf(to) := -/? 



E 

(i,j)eB 



N 

~EE^ 

1=1 Si=±l 



1 + rriiSi 



and has to be minimized. The Hessian is simply 



(^mf)ij 



d 2 F M F 
drriidmj 



= —(3Jij + Si. 



1 



1 



(2) 



Note that m = is a stationary point of the MF free 
energy — not necessarily a minimum — which we also call 
the paramagnetic solution. 

The stability of the paramagnetic MF solution implies 
the stability of the paramagnetic Bethe solution, as we 



shall now demonstrate. Define A := Ab^ — Amf', we have 
to show that A is positive semidefinite. Substituting QJ, 
(J2J and the relation £y = tanh(/3Jy ): 



A, 



-^r + arctanh^ + % ]T 
1 ? « fe=i 1 ^fc 



JV 



t2 



Note that A is symmetric and that all its diagonal entries 
are nonnegative. Furthermore, A is diagonally dominant, 
i.e. for all i = 1, . . . , N; 



N 



Au-^AA^Q 



3 = 1 

which immediately follows from the following inequality: 

~2 



1-x 2 



arctanh x 



1-x 2 



> 



that holds for all x E (—1,1). A simple application of 
Gerschgorin's Circle Theorem yields the desired result. 

Before we embark on spin-glasses, let us first shortly 
discuss the purely ferromagnetic case, where we take all 
nonzero interactions to be equal and positive, i.e. J = M . 
Both approximations are stable for high temperature (i.e. 
small 0) but break down at some critical /3 C , where the 
Hessians develop negative eigenvalues and the unique 
minimum of the free energy splits into two minima. For 
MF, this critical /3* /F is given by = 1/Ai, where Ai 
is the principal eigenvalue of the adjacency matrix M, as 
easily follows from J3J). Since d < Xi < A, we immedi- 
ately get the following bound on the ferromagnetic MF 
critical temperature: 



1 



MF 



< 



1 



For the Bethe approximation, the critical value (3^ at 
which Ase develops a negative eigenvalue can be shown 
to satisfy the following bound: 



1 



< 



1 



< tanh /3f e < 



A - 1 ~ Ai - 1 



1 



d-1 



In practice, for the graph topologies that we investigated, 
the critical values j3^ IF and j3f e differ only slightly. This 
is not the case for spin-glass like interactions, where the 
Bethe approximation clearly outperforms the MF ap- 
proximation, as we shall discuss shortly. 

In the following, we take the nonzero interactions 
{Jij | Mij = l,i > j} to be independent Gaussian ran- 
dom variables with mean and variance 1. We first dis- 
cuss the typical single- instance behavior. Fig.QJa) shows 
how the minimal eigenvalues of the stability matrices 
Amf and As e typically depend on the inverse tempera- 
ture (5. Varying the graph topology only scales the axes 
(except for tree-like or even sparser graphs), but qualita- 
tively the picture remains the same. For the Mean Field 
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FIG. 1: Transition from paramagnetic to spin-glass phase for 
an ER graph with TV = 100 and d — 20 and Gaussian inter- 
actions. From left to right: (a) minimal eigenvalues of Amf 
(solid) and of Ag e (dashed); (b) Edwards- Anderson order pa- 
rameter qEA\ (c) number of Belief Propagation iterations be- 
fore convergence. 

method, the paramagnetic solution becomes unstable for 
j3 > (3f F : = sup{/3 > 0;Amf > 0}. For the Bethe ap- 
proximation however, we typically find that with increas- 
ing (3, the smallest eigenvalue A m i n (As e ) first decreases 
until it becomes approximately zero at some critical (if e , 
after which it starts increasing again. At this critical (3f e , 
which we define as 

(if e := argminA min (A Be ), 

/3>0 

the paramagnetic solution is marginally stable. Our nu- 
merical experiments show that in the thermodynamic 
limit N — > oo, the minimal eigenvalue X m in{^Be) eval- 
uated at (3f e converges to 0. Furthermore, Monte Carlo 
simulations using the Metropolis algorithm show that the 
Edwards- Anderson parameter qEA '■= -h Yli=i ( s i) 2 be- 
comes positive at the transition (see Fig.^b)). We there- 
fore interpret this marginal instability as a phase transi- 
tion to a spin-glass like state [2fj|. 

In the remainder of this article we study the depen- 
dence of the Bethe and MF critical temperatures on the 
graph topology and its size N. Until now, everything is 
valid for single instances (consisting of a specific choice 
of G and of J), which is obviously very useful in ap- 
plications. In the following we will average over graph 
instances and over interactions in order to get an average- 
case analysis. We start with the uniform degree random 
graphs introduced and studied by Erdos and Renyi 
(ER graphs in short). The ensemble consists of graphs 
with N vertices, where each pair of vertices is indepen- 
dently connected with probability p. The degree distri- 
bution is approximately Poisson for large N and the ex- 
pected average degree is (d) = p(N — 1). The resulting 
model is also known in statistical physics as the Viana- 
Bray model of diluted spin-glasses [llj . Numerical results 
are shown in Fig. [2a and 0b. 

For N — ► oo, we can state rigorous results about the 
scaling behavior of the MF critical temperature. There 
are (at least) two ways to take the limit N — > oo, which 
result in different scaling behavior: the sparse (or finite 



connectivity) limit in which we fix the average degree 
d and let N — > oo, and the dense (infinite connectiv- 
ity) limit in which we fix p and take N — > oo. In the 
dense limit (Fig. 0a), (if IF ~ 1/2V5, which follows 
from a result of Khorunzhy [2l| stating that the prin- 
cipal eigenvalue of J is asymptotically equal to 2-^/Np, if 
d(N) > logiV. On the other hand, for d(N) < log AT, 
the spectral norm of J is asymptotically much larger than 
\fNp, which implies that the MF (if IF converges to zero 
in the sparse limit (c.f. Fig. 0b). 

For the Bethe approximation, we were not able to ob- 
tain rigorous results about the scaling behavior of fif e , 
but our numerical results turn out to be in agreement 
with the known phase boundary derived using the replica 
method. In the following equation is derived for 
the critical inverse temperature (i]( B corresponding to 
paramagnetic-spin-glass phase transition: 

J tanh 2 (^ s a;) dP(x) = 1 (3) 

where P is the probability distribution of the weights, 
which is Gaussian with mean and variance 1 in our case. 
For large d, this yields a critical inverse temperature of 
approximately (i^ B ~ 1/Vd- As illustrated in Fig. 0a 
and Fig. 0b, the average Bethe critical (if e perfectly 
agrees with the value j3]( B obtained by Viana and Bray, 
in the dense as well as in the sparse limit. This confirms 
our interpretation of the marginal instability as a phase 
transition to a spin-glass like phase. In sharp contrast 
with MF, the Bethe critical (if e converges to a positive 
constant in the sparse limit. In the dense limit the Bethe 
(if e is twice as large as the MF (3^ IF . 

We now present numerical results on Barabasi- Albert 
scale-free networks, for which no results have been pub- 
lished before as far as we know. A phenomenon often 
observed in real-world networks is that the degree dis- 
tribution behaves like a power law Q, i.e. the number 
of vertices with degree 8 is proportional to 8~ a for some 
a > 0. The first and intensely studied random graph 
model showing this behavior is due to Barabasi and Al- 
bert 0. The degree distribution has a power-law de- 
pendence for — > oo: the probability that a randomly 
chosen vertex has a particular degree 8 is proportional 
to 8~ 3 . The difference between the maximum degree A 
and the average degree d becomes quite large for the BA 
model compared with ER random graphs. A natural 
question to ask is what quantity will govern the spin- 
glass transition: the average degree d or the maximum 
degree A. 

In the dense limit (see Fig. 0c), we find (if IF oc \j2\fd 
and (if e oc 1/y/d. This is very similar to the ER case, up 
to a constant factor of order 1. Thus it appears that in 
the dense limit, the critical temperatures are almost inde- 
pendent of graph topology, except for the d-dependence. 
In the sparse limit (see Fig-Eld, note the rescaled vertical 
axis) , the two approximations show clearly different seal- 
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(a) ER dense (c) ER sparse (b) BA dense (d) BA sparse 




FIG. 2: Critical inverse temperatures /3^ IF (MF approximation, triangles) and f5^ e (Bethe approximation, squares) for the 
spin-glass transition vs. graph size N, averaged over interactions and over graph instances. From left to right: (a) ER, dense 
limit, p = 0.1 (the vertical axis is rescaled by l/Vd); (b) ER, sparse limit, d = 4; the solid line shows @Y B according to eqn. 10; 
(c) BA, dense limit, p — 0.1 (the vertical axis is rescaled by l/\fd); (d) BA, sparse limit, d = 10 (the vertical axis is rescaled 
by l/\/~A); the solid line is l/y/d and the dashed line is l/\/A. 



ing behavior: the MF critical 0^ apparently scales like 
1 / VA, wherease the Bethe critical /?* IF more closely fol- 
lows l/Vd. With increasing graph size, the difference be- 
tween both approximation methods becomes larger and 
larger. For the BA model, we have not found any theoret- 
ical results concerning phase transitions in the literature, 
but based on the analysis for ER graphs and on Monte 
Carlo experiments we believe that the Bethe approxima- 
tion correctly describes the phase transitions. 

In conclusion, we have applied the Bethe approxima- 
tion to spin-glass systems on random graphs. We have 
shown that the Bethe approximation is more powerful 
than the naive Mean Field approximation and that our 
approach agrees with previous results using replica meth- 
ods. The advantage of our approach is that it extends to 
arbitrary graph topologies and interactions and works for 
single instances, which is important for applications. In 
this light we would like to mention the connection with 
the Belief Propagation (BP) algorithm (2^, also known 
under the names Sum-Product algorithm and Loopy Be- 
lief Propagation. BP is a currently very popular algo- 
rithm used in diverse applications to minimize the Bethe 
free energy 0,H3|. As illustrated in Fig.QIc), and what 
we in fact have observed in all our numerical experiments, 
the BP algorithm converges only in the paramagnetic 
regime. At the transition to a spin-glass phase, the num- 
ber of iterations before convergence explodes. Details are 
beyond the scope of this Letter and will be explored in a 
forthcoming article. 

A possible generalization of this work would be the 
incorporation of local fields hi. Based on (preliminary) 
numerical experiments, we expect that adding local fields 
increases the Bethe critical inverse temperature /3,f e , 
thereby extending the paramagnetic regime (it is easy 
to see that the Mean Field critical fi^f increases). 

This research was supported in part by the Dutch 
Technology Foundation (STW). We thank Tom Heskes 
and Kees Albers for useful suggestions and interesting 
discussions. 



* Electronic address: j.mooij ©science. r u.nl| 
t Electronic address: b.kappen@science.ru.nl 
[1] R. Albert and A.-L. Barabasi, Rev. Mod. Phys. 74, 47 
(2002). 

[2] P. Erdos and A. Renyi, Publ. Math. Debrecen 6, 290 
(1959). 

[3] A.-L. Barabasi and R. Albert, Science 286, 509 (1999). 
[4] D. Watts and S. Strogatz, Nature 393, 440 (1998). 
[5] URL http : //elegans . swmed. edu 

[6] M. Paskin and C. Guestrin, Tech. Rep. IRB-TR- 

03-039, Intel Research Technical Report (2004), URL 

http : //www . intel-research . net/Publications/Berkeley/ 0728200 

[7] M. Mezard and R. Zecchina, Phys. Rev. E 66, 056I26 - 
(2002). 

[8] S. Wuchty, Mol. Biol. Evol. 18, 1694 (2001). 
[9] E. Kussell and E. Shakhnovich, Phys. Rev. Lett. 89, 
168101 (2002). 

[10] D. Sherrington and S. Kirkpatrick, Phys. Rev. Lett. 35, 
1792 (1975). 

[11] L. Viana and A. Bray, J. Phys. C: Solid State Phys. 18, 
3037 (1985). 

[12] M. Leone, A. Vazquez, A. Vespignani, and R. Zecchina, 
Eur. Phys. Jour. B 28, 191 (2002). 

[13] K. Tanaka, J. Phys. A 35, R81 (2002). 

[14] W. Freeman, E. Pasztor, and O. Carmichael, Int. J. Com- 
put. Vis. 40, 25 (2000). 

[15] R. McEliece, D. MacKay, and J. Cheng, IEEE J. Sel. 
Areas Commun. 16, 140 (1998). 

[16] H. Kappen, in Modelling biomedical signals (World Sci- 
entific, Singapore, 2002), pp. 3-27. 

[17] J. Yedidia, W. Freeman, and Y. Weiss, in Advances in 
Neural Information Processing Systems (2001), vol. 13, 
pp. 689-695. 

[18] M. Welling and Y. Teh, in Proc. of the Conf. on Uncer- 
tainty in AI (2001), vol. 17. 

[19] R. Baxter, Exactly Solved Models in Statistical Mechanics 
(Academic, New York, 1982). 

[20] M. Mezard, G. Parisi, and M. A. Virasoro, Spin glass 
theory and beyond (World Scientific, Singapore, 1987). 

[21] A. Khorunzhy, Adv. Appl. Probab. 33, 1 (2001). 

[22] J. Pearl, Probabilistic Reasoning in Intelligent systems: 
Networks of Plausible Inference (Morgan Kaufmann, San 
Francisco, CA, 1988). 

[23] T. Heskes, in Advances in Neural Information Processing 



Systems (2003), vol. 15, pp. 343-350. 



